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ABSTRACT 

Metacognitive activities were studied through the 
verbalizations of students during a learning process, to study the 
differences in the performance of good and poor students. The moment 
of verbalization, either at prompting marks or without prompts, and 
the instruction in verbalization, either directed or non-directed, 
were variables. Thirty-two girls and 32 boy^ in the first year of 
Dutch comprehensive school, half good performers and half poor 
performers according to school records, participated. Half were 
instructed to verbalize only at marks occurring during ^he learning 
of lists of English vocabulary words and Dutch translations; half 
were to verbalize throughout the learning process. Half received 
directed instruction and half did not. Thinking-aloud protocols were 
tape-recorded and analyzed. Analyses of variance indicated that the 
two variables "moment of verbalization" and "instruction" did not 
have any effect on performance. Both variables also had no effect on 
the total quantity of verbalization* Most students, however, could 
not restrict themselves to verbalizing only at the marking points, 
and most verbalized between marks as well. To keep verbalizations 
focused on the task, instruction was helpful. Good students 
apparently interrupted their recalling action during learning by 
checking and noting their own accuracy more often than did poor 
students. Six tables present study data. (SLD) 
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RECI5TRAU0H OF SELF-REmAUOH PROCESSES tllTH FIRST YEAR STUDEHTS 
OF A HAVO-m COHPREHEHSm SCHOOL BY HEANS OF THINKIHQ ALOUD 

FRAHK PX.M. or JOW 
University of Tilburg 

!• Introduction 

Although a lot of research on individual difference in nwtacognition nakes 
use of verbal reports, most are still ^measureinents at long distance call'' 
with a lot of noise on the line. It is rare that verbal reports consist of 
on-line thinking aloud during learning. Instead of on-line registration 
one is trying to get insight in childrens* nctacognition by derivations of 
thinking aloud, for exaiple by asking children which strategies they would 
use in hypothetical situations (Kreutzer, Leonard, Flavell, 1975; Trepa- 
nier, 1981) or by predicting their netoryspan (Yussen and Levy, 1978; Fla- 
vell, Friederichs and Hoyt, 1970; Markian, 1973). Another frequently used 
verbal report is of the retrospective kind. It is on these retrospective 
protocols that Nisbett and Wilson (1977) ground their conclusion that ver- 
bal reports "nay not interrogate meaorization of the cognitive processes 
that operated between stimulus and response but a lot of theorizing about 
the causal connections between stinulus and response" (p. 23}). 
Such noise or discrepancies between verbal reports and what people actual- 
ly do or think is inherent in verbalizing, which takes place before or af- 
ter the task and not during the task. Ericsson and 5i«on (1980) conclude 
that if people do not have to retrieve information from other sources than 
their short-term memory, there is no question of epiphenomenality. Studies 
on problfem solving (Duncker, 1935; De Groot, 1946; Newell and Simon, 1972; 
Elshout, 1976; Bretker, 19B1; Vermunt, 1984) have shown that thinking 
aloud is an excellent method for research on higher cognitive processes. 
But it is especially the development of self-report models (Ericsson and 
Simon, 19B0; Elshout, 1976; Breaker, 19B1) that lifted the thinking-aloud 
method out of its banishment from the scientific scene. 
But even if you decide on the basis of these powerful theoretical sub- 
structures to use thinking aloud as registration method of metacognitive 
processes, you will be faced with questions concerning the creation of 
circumstances under which people easily verbalize, and which have their 
impact on the efficiency of the protocols and their analysis as well. This 
is important because it still takes up an incredibly great deal of time. 
Tn order to dam the time devouring work of protocol analyses it is reali- 
r^tic ti put foreward the question if it is necessary that people express 
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all their thoughts if a researcher is interested in a part of the verbali- 
zing. What is the relevance of registrating that somebody has a party in 
the afternoon? Not only from the efficiency point of view it might be bel- 
ter to instruct people in such a way they only express relevant thoughts, 
but also for ,,e sdke of clarity towards people. With a more directed in- 
struction it is easier for people to understand what is asked by thinking 
aloud. Because of this clarity it might even be that verbalizations are 
stimulated. 

Stimulations are important because not everybody is an easy talker and it 
serves the researcher to have rich protocols. 

From the same point of view it is suggested (Olshavsky, 1976/77) to put 
marks in a text. But it is still the question if such remembering points 
have a stimulating or restraining influence on verbalizations when people 
have to learn vocabulary. 

To answer this question, an experiment was conducted in which the moment 
of verbalization (marks/no marks) and the instruction (directed/non direc- 
ted) of what to verbalize were manipulated. 

However, the first interest of the study concerns the question: What is 
the difference in the equipment between good and poor students that makes 
good students be better performers? 

A lot of studies (Flavell, 1979; Flavell, Wellman, 1977; Brown, Deloache, 
197B; Brown, Palinscar, Armbruster, 19B4; Brown, Armbruster, Baker, 1986) 
show a difference jn the metacognition. Training metacognitive activities 
(Brown et al., 1980; Palinscar, Brown, 19B3) also shows to have a great 
influence on the performances of poor students* This suggests that meta- 
cognition is a more critical variable in this matter than strategy defi- 
ciency. Therefore metacognitive activities were registrated on tape by the 
method of thinking aloud and Student Level was manipulated. 

2. Procedure 

32 girls and 32 boys, of whom one half were ''good" and one h:^lf were 
"poor" performers according to admission criteria of their schools and 
their performance on a national examination, participated in the experi- 
ment. Students were trained in thinking aloud during learning - or "lear- 
ning aloud" - in two introductory sessions. During these sessions half of 
the students were instructed to verbalize everything they thought of du- 
ring learning, and the other half were instructed to verbalize only those 
thoughts which were related to self-regulating activities. Both groups got 
a list of 44 examples of thought expressions which were related to self- 
regulation. 

The list that was given to the students of this first group (pUo were told 



to vflrbalixe everything) m«s extended wit.h 17 thought expressions that 
Mere not related to self regulation activities. 

In the Mee'. after the introductory sessions each student had tMO learning 
sessions on two different days, in which they got exactly 15 Minutes to 
learn a list of 40 English words and their Dutch translations. Before each 
learning session the students were told that after the learning session 
they had to give the Dutch translation of 20 English words » randoaly ex- 
tracted froa the learning task, and the English translation of the 20 
Dutch words, (that reaained froa the learning task). Furtheraore they were 
reainded of what they had to verbalize. They were instructed wether they 
had to try to learn aloud constantly or only when they encountered a mar- 
king- point in the vocabulary list. Each learning session started with li- 
stening to the pronunciation of the English words which were recorded on 
tape. 

Ihe variable Instruction (directed/non directed) was a between subjects 
factor and assigned to a subject during the total experiaent. Good and 
poor perforaers were counterbalanced over the variable Instruction as was 
the variable boys/girls. Because the variable "Hoaent of Verbalization" 
(thinking aloud at aarking points or constantly) was a within variable, 
two parallel vocabulary lists were necessary, which were counterbalanced 
over the variable Moaents of Verbalization. The order of presentation of 
the two vocabulary-lists as well as the order of presentation of the two 
levels of the variable Moaent of Verbalization were counterbalanced over 
subjects. 

The thinking-aloud protocols were recorded on tape, typed out literally 
and analyzed into aeaningful units (Wouters and de Jong, 1982). Every 
meaningful unit contains one process and is not restricted to the liaits 
of a sentence. These aeaningful units have been analyzed in five general 
categories: trans: foraing, orientation^ aonitoring, directing and testing. 
Each category consists of several operations . Another category, stimula- 
tion, deals with the experimenter's stimulating remarks to verbalize and 
other remarks by the experimenter and student. 

The process analyzinr/ scheme is a modification of the scheme which is de- 
veloped by Vermunt, Lodewijks and Simons (19B6). Modifications are based 
on analyzed protocols of a pilot study as well as on the first analyzed 
protocols of the study in question. 

3. Results 

Data were first subjected to some analyses of variance which concerned a 
checking of the used counterbalancing. There was no significant effect on 
the learning achievement of the students that resulted from the order of 



presentation of the two vocabulary lists. Whether a list of words con- 
tained marks or no aarks had also no significant effect on the students' 
learning achievements. Secondly, MANOVAs were carried out to determine 
whether the achievement differed systematically between the conditions 
Harks and No Marks. The factors in these MANOVAs were Instruction, Student 
Level, Moment of Verbalization and Delay with a repeated measurement on 
the last two factors. Dependent variables were the performance on the four 
translation tests (from English to Dutch and from Dutch to English). Stu- 
dent Level had a significant main effect (r(2.59) r 6.59; p< 0.05). Delay 
also had a significant main effect (r(2.59) = 82.03; p< 0.0001). The Mo- 
ment of Verbalization had no main effect on the achievement. There was no 
main effect of Instruction although some univariate analyses showed a 
significant effect. A close look at the means (see table 1) , however, 
shows an unsystematically crossed effect. There were no significant inter- 
actions. 



Instruction | First test || Retention test 





1 1 

1 1 


E - D 


1 D-E 


II 


E - D 


1 D - E 1 


|Non-directed | 

1 1 


13.20 


1 12. B4 




10.9 


1 10.29 1 

1 1 


lOirected | 


14.31 


1 14.22 


II 


12.26 


1 11'41 1 


(Difference | 

1 1 


1.11 


1 1.3B 


II 


1.36 


1 1.12 1 



^^^^^ 1= Performance means of variables Instruction and Delay 



'E - D = English - Dutch; D - E = Dutch - English) 

Furthermore, two separate series of ANOVAs were carried out with the fac- 
tors Instruction, Student Level and Delay. One significant interaction 
between factors Delay and Student Level was found concerning the variable 
Translating English into Dutch under the condition No Harks (F(7.60) = 
4.49| p< 0.04) of the variable Moment of Verbalization. 
After these analyses which concerned the impact of the experimental va- 
riables on students' performance an ANOVA was conducted to determine the 
conditional effects on the total amount of verbalizations. Factors in this 
ANOVA were Instrurtion, Student Level and Moment of Verbalization with a 
repeated measurement on the last factor. Student Level had a significant 
effect on the amount of verbalization (F(1.60) = 9.02; p< 0.004). 
In order to determine systematical effects of conditions on the amount of 
verbalized processes, when these were divided into the six analyzing cate- 
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gories. ■ HANOVA mis conducted with the fectors Instruction, student Le- 
vel, Mownt of Verbalizetion end the cetegoriee TrensforBing, Orientatio, 
Monitoring, Directing, Testing end Stiiwleting as variates. The factor Ho- 
■ent of Verbalization mbs treated as a repeated aeasureMnt. Student- Le- 
vel (F(6.55) = 5.67; p< O.OOOl) and Noaent of Verbalization (Fv6.55= 5. 13) 
P< 0,003) had significant Main effects. 

When we look at the univariate analyses concerning the factor Student Le- 
vel there were significant affects on the variates Honitoring (F(1.60)- 
12.97; p< 0.0006); Testing (F(l.60)= 20.47; p< 0.0001) and Stiiulating 
(F(4.60)z 4.60; p< 0.04). (For Means see table 2). 



II HoMent of 



Categories j student Level j] Verbalization 



« -w.w. ,, '«^4 UOAA^at AUII 

1 ' III 1 




1 good 

1 


1 poor 
1 




no aarks { 

1 


Marks 1 


jlransforiing 


1 

1 32.6 


1 

1 30.6 
1 


II 
II 


1 

34.9 1 


2B.3 1 


jOrientation 


1 2.4 


1 1.8 


II 


2.0 1 
1 


2.2 1 


1 Monitoring 


40.7 


1 24.7* 
1 


II 


33.2 1 
1 


32.2 1 


iDirecting 

1 1 


17.3 


1 14.4 




16.3 1 
1 


15.1 1 


1 Testing 


33.2 


1 19. I« 




29.9 1 
1. 


22. 4» 1 


1 Stimulating | 

1 i 


6.5 


1 10.4 




5.9 1 


10. 9» 1 



lablLli Means per category for the factors Student Level and 
Monent of Verbalization 

• Significant difference between condition levels according 
to Bonferoni adjustments 

When we look to the univariate analyses concerning the within factor Mo- 
ment of Verbalization, there were significant effects on the variates 
Transforming (F(1.60)= 7.41, p< o.Qi). Testing (K(l. 60)= 12.04; p< Q qqi) 
and Stimulating (F(I.60)= 15.45, p< 0.0002). (For means see table 2) 
Instruction had no significant effect on the amount of verbalized proces- 
ses. There were no significant overall interactions. Only the interaction 
between Moment of Verbalization and Instruction in the univariate analysis 



for the variate Orientation was significant (F(1.60)= 5.78; p< 0.02) Cell 
Means are given in table 3. r /. uc** 



1 Instruction 
i 


1 No Mrks I 

1 1 


Marks | 
1 


1 Non-directed 
1 


1 2.9 1 
1 1 


1.5 1 

1 


1 Directed 


1 1.2 1 


2.9 1 
1 



MiLii Cell Means for the factor Instruction and HoMent of 
Verbalization concerning the category "Orientation" 

A HANOVA with the factors Instruction, Student-Level, MoMent of Verbaliza- 
ion as within factor, and the subdivisions of the category StiMulating as 
its was carried out to exaMin which interventions r- interruptions 

1 h \T °f Verbaliza- 

tion had a significant Main effect (F(5.56)= 3.92; p< 0 005) 

A further inspection of the univariate analyses (see table i) established 

that only for the variate General ReMarks no effect of Harks or No Mar s 

-ore stimulation took plac students iZl 



{Variables 


1 No 
1 marks 

1 


1 Harks 
1 


1 df 

1 

1 


1 F 

1 

1 


1 P 

1 

1 


1 remarks experiment 


1 1.6 


1 3.2 
1., 


1.60 


1 7.88 


1 0.0067* 
1 1 


Igen. remarks subj. 


0.1 


0.2 

1 


1.60 


1 0.48 


0.4920 1 
1 


Istim. to verbalize 


2.8 1 


4.9 1 


1.60 


14.90 


0.0003* 1 


questions subject | 


0.3 1 


0.9 1 


1.60 1 


10.48 1 


0.0020* 1 


reactions subject | 

ah la A. u ^ 


I.O 1 

i 


1.8 1 


1.60 1 
1 


4.24 1 


0.0439 1 



I3ble_4: Means and F-values for the variates of thel^^ 
"Stimulation" for the conditions No Marks and Harks 
» Significant effects according to Bonferoni adjustment 

There .as also a significant overall interaction between the within faci* 
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and Student Level (r(5.56)s 4*99; p< 0*001)* 

Of the univariate interaction between Honent of Verbalization and Student 
Level Ihere Mas only a significant interaction for the variate Stinula- 
tions to Verbalize (F(l*60)s 10*32; p< 0*003)* 

A close look at the means (see table 5) shows that especially good stu- 
dents needed more stiaulations in the Harks condition* 



1 Student Level | 

1 1 


No Marks | 


Marks | 


ICood 1 

i 1 


l.B 1 


5.6 1 


|Poor 1 

1 1 


3.8 1 


4.2 1 



Table 3: Cell neans of Stiaulation to Verbalize for 



the factor Honent of Verbalization 

As mentioned before, Student Level and Moment of Verbalization had main 
effects on the five process categories in which the analyzed processes 
were placed* If we view the univariate analysis, the factor Student Level 
had significant effects on the categories Monitoring (r(1.60)= 12.97; p< 
0.001), and Testing (F(L60)= 20.47; p< 0,0001). 

In order to get aore insight into thp differences between the processes 
that took place during learning witti the two Student Level groups, two MA- 
NOVAs for each level of the factor Moment of Verbalization were carried 
out« The factor in these HANOVAs was Student L^vel and the variates were 
the processes of the category Monitoring or Testing. 
In both of the conditions No Marks and Harks means differed significantly 
for good and poor students for the processes "noting of a negative inter- 
result", "noting of 3 positive inter- result" and "recalling", (see table 
6). 



1 1 
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Con- 1 
di- 1 
tion 1 


Process | 
category 1 


Processes 


mean | 

stud* lev. 1 
good 1 poor 1 


r 1 


Df 1 


P 




Honi- 1 


noting a pos* 
inter-result 


14*5| 5*7| 
1 1 


12*56| 

* 


1.6oi 

- 


0.0394 


No 1 


toring 1 


noting a neg* 
inter^result 


i9.e|io*i 


13.50 


1*60| 


0.0005« 


marks 1 




recalling 


28*6|16*8 
1 1— 


12*27 
► 


1.60| 


0.0009« 




Testing 


recalling and 
writing 


2.4| 0.1 


5*43 


1.60| 


0.0231 






correction 


1 2.3| 1.4 


4.70 


1*60| 


0.0341 




Moni- 


noting a neg. 
inter-result 


|13.6| 6.2 


7.14 

■ 


1.60| 
— • 


0.00097 


Marks 1 


toring 


noting a pos. 
inter-result 




|16*0|10.0 


6.66 


1.60| 


0.0123 


Testing 


recalling 


|22.0|12.2 


110.90 


|1.60| 


0.0016* 



Table 6: Significant mean differences between good and poor Student Level 
for the categories "Monitoring" and "Testing" 
« Significant according to Bonferoni adjustment 

4* Discussion and conclusion 

Taking the results of the analyses concerning the experimental variables 
"Moment of Verbalization" and "Instruction" together, it is clear that the 
two experimental variables "Instruction" and "Moment of Verbalization" do 
not have any effect on the performance. Both variables have no effect on 
the total quantity of verbalizations, either. However a more detailed ana- 
lysis of the da^a reveals some differences for the two levels of the 
variable "Moment of Veraalization". The amount of stimulations which are 
necessary to remind students to verbalize their thoughts in the condition 
MarLfQ iq csfrikinn. Arid tn this th»t fflnst students couid not restrict them- 
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selves to verbalize only at the aarking points. It frustrated then to in- 
terrupt the< . learning at the Mrks. So almost ill students autonatically 
verbalize their thoughts during the tiu^ between two »arks as well. The 
real difference betneen the two conditions No Harks and Harks is that stu- 
dents had to verbalize at narks, and stinulations to verbalize were only 
given when a nark was encountered. 

As students found it nore natural to think aloud constantly, and because 
less interruptions of the learning process were necessary and because the 
verbalizations that took place under the condition No Harks were certainly 
not fewer or inferior, it is this situation which has the preference. 
If renarks like '^it is ^ery hot here today'' are not of interest, it is 
preferable to give a nore generally directed instruction in which students 
are not pinned down in what to say but are given an inpression of what is 
intended by the tern think ing-aloud. 

The results 'Concerning the difference between good and poor students con- 
forn to the findings of Sinons, Vernunt, Lodewijks (1963). They also found 
great differences in nonitoring and testing activities between good and 
poor students. 

However, it is not so that the results can be interpreted in this way that 
good students recall nore than poor students, but their recalling activity 
during learning is nore often interrupted by activities of checking and 
noting whether the words they recalled were correct or incorrect. In other 
words; good students keep nuch nore an eye on the results of their lear- 
ning process by checking whether they know the learned words or not* 
Another striking thing is that orientation as to how nany words and what 
words had to be learned in such a short tine, hardly took place. Also 
hardly any adaptations took place of one's study strategy to the remained 
study tine» Finally it is remarkable that there is no difference in trans- 
forming activities between the groups of poor and good students. 
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